Overcoming the pitfalls and perils of algorithms: A classification of machine learning biases and mitigation methods

2.1. Artificial intelligence and machine learning in marketing

The adoption of AI technologies across organizations is growing rapidly, and firms are increasingly recognizing the practical opportunities arising from their ability to perform human-like tasks such as learning autonomously or making decisions based on large datasets (Huang & Rust, 2021). In particular, AI has witnessed impressive breakthroughs in image recognition, speech processing, autonomous driving, and many other tasks typically considered to require human-level intelligence (Davenport et al., 2020). Behind much of this breakthrough is ML, which has become the main paradigm of contemporary AI research (Wang, Ryoo, Bendle, & Kopalle, 2021).

ML is a vast and rapidly evolving field, encompassing a wide range of methods for addressing diverse tasks (Ma & Sun, 2020). Despite this variety, the typical application logic of ML in marketing is summarized in Fig. 1. Data is generated from the relevant population which is used to train a predefined ML model that often classifies observations or optimizes a predefined outcome, and its predictions then trigger marketing decisions and actions. For example, Netflix generates data from the viewing behavior of all its customers, uses this data to train a recommendation algorithm, whose predictions then trigger individual movie and series recommendations for all its customers. Building upon this logic, a diverse set of ML methods, such as support-vector machines (Cui & Curry, 2005), topic models (Tirunillai & Tellis, 2014), ensemble trees (Yoganarasimhan, 2020), and deep neural networks (Liu et al., 2020), have been used in marketing for making predictions that trigger decisions and actions.²

1. Download : Download high-res image (193KB)
2. Download : Download full-size image

Fig. 1. Conceptual Approach: Using Machine Learning for Marketing Problems.

2.2. Reasons for machine learning biases

The decision-making of algorithms is very different to human decision-making in two important ways: algorithms are extremely literal and they are black boxes. First, while humans understand soft goals and trade-offs, algorithms will pursue a specified objective single-mindedly (Luca et al., 2016). For example, Ukanwa and Rust (2020) show that for loan decisions, discriminatory results can occur even if there is no bigotry programmed into the algorithm because the algorithm only seeks to maximize profit. Second, algorithms are black boxes in the sense that they can often form predictions with great accuracy, but they do not provide causes or reasons for an event. Thus, algorithms often lack interpretability, in terms of having a transparent model structure and clear linkage between variables (Ma & Sun, 2020).

Moreover, ML algorithms differ substantially from deterministic, rule-based algorithms that have been used in the past for decision support in the organizational context (Wang, Ryoo, Bendle, & Kopalle, 2021). ML algorithms, such as neural networks, follow a probabilistic approach in which decisions are not made by following programmed rules but by learning patterns from historical data and applying these to new input data. The decision support from ML algorithms is provided in the form of probabilities, leading to different levels of uncertainty and therefore increased susceptibility to systematic biases. For instance, Lambrecht and Tucker (2019) have shown that gender bias can occur without any conscious (or unconscious) attempt to produce a biased outcome—using only an unbiased algorithm.

Finally, a series of subjective choices must be made in the process of any ML project, and all these choices may introduce biases and lead to unwanted outcomes. For example, considering the logic of Fig. 1, not all relationships within the relevant population necessarily generate data, human coding might determine the data generation process, and the impact of the ML model may reinforce certain patterns in the data generation. As a result, the data does not represent the “whole” relevant population because not all observations and relevant variables are recorded. Even if the data is perfectly unbiased, the decision on how to build and train the model can introduce biases (e.g., selection of unsuitable variables and over- or underfitting during the model training). Even if one assumes the resulting ML application is free from bias introduced through data or design decisions, an inappropriate context of use may nevertheless lead to a bias.

Research in business, marketing, computer science, psychology, and sociology has started to consider ML biases (e.g., De Bruyn et al., 2020, Guha et al., 2021, Huang and Rust, 2021, Puntoni et al., 2021). However, to date only a few scattered articles provide a more detailed examination of certain ML biases and mitigation methods (e.g., Baeza-Yates, 2018, Mehrabi et al., 2019, Silva and Kenney, 2019, Suresh and Guttag, 2019), and across these articles the respective terminologies differ substantially. We unite these dispersed perspectives with a systematic, interdisciplinary review that summarizes potential ML biases throughout the full ML project lifecycle.

3.1. Process phases of machine learning projects

Fig. 2 displays the six phases of the CRISP-DM process model that can be used to plan, organize, and implement an ML project (Martínez-Plumed et al., 2019). Anticipating our findings, we also embed the eight ML biases from our review into this figure. Published in 1999 to standardize data mining processes across industries, the CRISP-DM has since become the most common process model for data mining, data analytics, and data science projects.

1. Download : Download high-res image (365KB)
2. Download : Download full-size image

Fig. 2. Process Phases and Potential Biases of Machine Learning Projects Note: The cross-industry standard process for data mining process model is based on Wirth and Hipp (2000).

The initial business understanding phase focuses on understanding the ML project objectives and requirements from a business perspective, then converting this knowledge into an ML problem definition and a preliminary plan designed to achieve the objectives. The data understanding phase starts with initial data collection and proceeds with activities that enable the researcher to become familiar with the data (i.e., describe data, explore data, and verify data quality). The data preparation phase covers all activities needed to construct the final dataset from the initial raw data that will be used for the ML algorithm. Tasks include observation and attribute selection as well as transformation and cleaning of the data. Several ML techniques are then selected in the modeling phase and applied to the prepared dataset, before their parameters are calibrated to optimal values. The evaluation phase determines which ML model best meets the success criteria and reviews the work accomplished. Before proceeding to the final deployment of the ML model, it is important to thoroughly evaluate it and review the steps executed to create it, to be certain that the ML model adequately achieves its objective. Depending on the requirements, the deployment phase can be as simple as generating a report or as complex as implementing a self-learning algorithm for organizational decisions (e.g., marketing spending per channel).

3.2. Data collection and analysis

We conducted a systematic, problem-centered literature review to integrate existing knowledge about ML biases through the conceptual lens of the CRISP-DM model. First, different types of ML biases are identified and consolidated into distinct categories. Second, possible mitigation methods that address these biases are grouped. Third, both ML biases and mitigation methods are incorporated into the different phases of the CRISP-DM model.

To identify relevant articles for our review, we performed a systematic keyword search in EBSCO, AIS Electronic Library, ACM Digital Library, ScienceDirect and Emerald Data Base with a focus on leading journals in Business, Marketing, and Information Systems research. This initial search revealed 61 articles in total. We then conducted a screening, and only considered articles that are peer-reviewed and/or conference proceedings, define at least one relevant ML bias, and explain at least one relevant identification or mitigation method. After applying these criteria, 24 relevant articles remained. We next performed forward and backward search on the relevant articles using Web of Science. This step added 31 articles based on the same inclusion/exclusion criteria. Finally, senior scholars in the research domain were contacted for additional relevant literature, and 13 articles were added. In total, 68 articles were included in the literature review. We provide an overview of all articles in the appendix. The literature search process is depicted as a flowchart diagram in upper part of Fig. 3.

1. Download : Download high-res image (746KB)
2. Download : Download full-size image

Fig. 3. Selection Criteria and Evaluation Framework.

In the data analysis phase, the different biases and mitigation methods were extracted from the articles. Because different synonyms exist in the literature for the same type of bias, all biases were then descriptively synthesized based on their mechanism. This allowed us to code the biases into eight distinct categories, as summarized in Fig. 2: social bias, measurement bias, representation bias, label bias, algorithmic bias, evaluation bias, deployment bias, and feedback bias. To derive the eight distinct biases, we followed the method for taxonomy development by Nickerson et al. (2013). Specifically, we used the empirical-to-conceptual approach meaning that we started with empirical data clusters and deductively conceptualized the nature of each cluster afterwards. The meta-characteristic used for distinguishing the identified biases is the origin (i.e. the occurrence in the CRISP-DM process model) and cause of the bias (i.e. in the data, through humans, in the algorithm, etc.). We stopped iterating when the objective (no bias was merged with a similar bias or split into multiple biases in the last iteration) ending conditions were met.

The taxonomy meets subjective ending conditions as well. The taxonomy is concise, robust, comprehensive, extendible, and explanatory. With eight dimensions, it is meaningful without being overwhelming (concise). The dimensions can differentiate among objects (robust) and classifying random samples of objects within the domain (comprehensive). New dimensions can easily be added if needed (extendible). As the identified characteristics are sufficient to understand the differences among types of bias, the taxonomy is also explanatory. The eight distinct biases were then assigned to the phases of the CRISP-DM process model by matching the mechanism that leads to a specific bias to the tasks in each of the phases (see Fig. 2).

We also identified 24 mitigation methods, allocated them to the respective bias they address, and assigned them to the phases of the CRISP-DM process model. The allocation of mitigation methods was independently conducted by two researchers to enhance reliability. This process involved the extraction and description of each identified mitigation method and an assessment of how the method was suited to addressing the identified types of bias. In most of the analyzed articles the bias-mitigation method relationship was evident. For each mitigation method, and particularly in case of ambiguities, the research team carefully reviewed how each mitigation method affected elements in the CRISP-DM process and how it mitigated bias. In the next step, each bias mitigation method was assigned to a particular process phase in CRISP-DM. The lower part of Fig. 3 provides an overview of the allocation process.

4.1. Overview of machine learning biases

A measurement bias is introduced if chosen features and labels are imperfect proxies for the real variables of interest. When defining the target variable and necessary features for the ML problem, researchers may choose imperfect proxies for the true underlying value or include protected attributes (Mullainathan & Obermeyer, 2017). Protected attributes refer to attributes such as race, gender, or ethnicity that partition a population into different groups that should be treated equally. Using protected attributes as proxies for other features of interest may result in a discriminant or inaccurate classifier. But even if the protected attribute is excluded, the discriminant effect can still exist due to the redlining effect, which states that protected attributes can correlate with non-protected attributes and still bias the outcome (Corbett-Davies and Goel, 2018, d’Alessandro et al., 2017). For example, in a crime prediction application, the feature “number of arrests” is used to predict future criminal activity. Assuming African American and Caucasian defendants commit the same number of drug sales, they have a similar true risk. However, in minority neighborhoods with heavier policing, African American defendants are more likely to experience drug arrests. Despite the similar true risk, the ML model would therefore classify African Americans as a higher risk than Caucasians (Angwin et al., 2016).

A representation bias arises when the input data is not representative of the relevant population, which leads to systematic errors in model predictions. This bias relates to a lack of representability either evoked by the specific time of data collection or the specific sample. First, a representation bias can occur when the training data is no longer representative of the relevant population when the model is deployed. For example, data can be disturbed by one-time phenomena. ML algorithms built for credit card applications use historical data regarding the probability of a credit default. In case of an unsuspected event during the data collection, such as a natural catastrophe in a certain area, people might not be able to pay back their debts. Therefore, applicants from this area will most likely be classified as potential defaults (Baer, 2019). Second, a representation bias can also occur when the training data is wrongly sampled. Such a bias emerges if the distribution of the sampled population differs from the “true” underlying distribution in the relevant population. This over- or underrepresentation can have several causes, including difficult or expensive availability of required data. The algorithm consequently fails to make good predictions for the overall population (Lan et al., 2010).

A label bias arises when training data is assigned to classes or labels that systematically deviate from the underlying truth categories. Researchers often face the difficulty of deciding which available label best applies to the present data, and existing labels may also fail to precisely capture meaningful differences between classes (Barocas & Selbst, 2016). Moreover, due to ambiguity and cultural or individual differences, labels might systematically deviate. One such example is the assumption that a certain number of pictures are to be labeled as “wedding”. A person that is brought up in western culture will likely only label pictures with brides in white dresses and grooms in dark suits as “wedding”, and thus fail to label pictures of an Indian wedding, with its colorful dresses and special decorations, as a wedding (Baer, 2019).

An algorithmic bias occurs when inappropriate technical considerations during modeling lead to systemic deviation of the outcome. This occurs when formulating the optimization problem when researchers make data and parameters amenable to computers (Dwork et al., 2011, Friedler et al., 2019). The resulting ML model may fail to treat groups fairly when the probability of misclassification, i.e., false-positive and false-negative rates, are distributed unequally among groups (Bellamy et al., 2018, d’Alessandro et al., 2017). For example, minorities exhibited a higher false-positive rate than majority groups in COMPAS, a predictive policing application that assesses the risk of crime recidivism (Chouldechova, 2017). Such a misclassification can also result in service discrimination for certain individuals (Ukanwa & Rust, 2020).

An evaluation bias takes place when a non-representative population or inappropriate performance metrics are used to evaluate the model. ML models are often tested on the same benchmark data to allow for an objective comparison. However, if the benchmark itself is not representative, models could be preferred that only perform well on a subset of the relevant population (Suresh et al., 2018). Thus, choosing the wrong benchmark data can lead to overlooking potential biases. For example, if a facial recognition algorithm is trained on a dataset with underrepresented dark-skinned females and is tested on a similarly unbalanced benchmark, the bias will remain unrecognized (Suresh & Guttag, 2019).

A deployment bias arises when the model is used and interpreted in a different context than the one it was built for. This bias occurs because no ML algorithm operates fully autonomously; rather, an ML algorithm requires the input of human decisions when it is deployed. The researchers deploying an algorithm may differ from those that built it: They may have a different knowledge base or values and interpret the algorithmic output according to their internalized biases (Bellamy et al., 2018, Chouldechova, 2017). For example, certain risk assessment ML models are built to predict the likelihood of a criminal committing a future crime. However, in practice, these models are often used in different contexts, such as determining the length of defendants’ sentences (Collins, 2018).

A feedback bias arises when the outcome of the ML model influences the training data such that a small bias can be reinforced by a feedback loop. It emerges when the output of the ML model is used as a new input, and the algorithm is refined over time (e.g., through re-training). If the outcome of the ML model has an influence on subsequent training data, an initially small bias may be potentially reinforced through a feedback loop (Bellamy et al., 2018, Martin, 2019). For example, once a certain piece of content attains a good ranking according to a rating algorithm based on the number of times it has been clicked, it will affect the position and the promotion of this content, thus leading to even more clicks. Consequently, a reinforcing feedback loop is created and can lead to decreased user satisfaction when unwanted content is promoted (Baeza-Yates, 2018).

4.2. Overview of mitigation methods

5.1. Social bias: A customer reward initiative that ignores loyal customers

For the bakery’s 10th anniversary, the marketing team initiated a reward initiative in which particularly loyal customers were to receive a voucher. In determining the criteria for the reward scheme, a classification algorithm was applied to the bakery’s customer sales data. It transpired that the features “average spending” and “frequency of visits” were important for the classification of customers who generate most revenue and should therefore be rewarded. Due to the classification, mainly business travelers were rewarded because of their relatively high spending and frequency of visits. However, they are not very loyal in reality. In contrast, teenagers from nearby schools who made frequent visits but had lower spending were disadvantaged even though they are very loyal customers.

The bias emerged because the data-driven process neglected the fact that teenagers were disadvantaged in the reward initiative compared to higher-earning business travelers because of their relative lack of purchasing power. Although the data used for classification led to an economically and mathematically correct classification of customers, it is questionable whether it was socially desirable to disadvantage teenagers who are loyal customers. Such a bias also entails reputational risk and may possibly jeopardize future business potential since teenagers represent a customer group with increasing customer value as they grow older. An effective method to counter such a social bias is rapid prototyping. The bakery should have deployed the ML-model for a short period of time only and asked the bakery staff for feedback on the selection of customers. Another way of mitigation would have been if the staff had handed out vouchers to their most loyal customers to compare the ML-generated selection with the experience-based selection of customers.

5.2. Measurement bias: Capturing good weather effects

Cakes have a high margin for the bakery but also a limited shelf life. Consequently, production and sales quantities must be carefully planned, as both over- and under-capacities are costly (i.e., they result in loss of sales or discarded products). The management knew from experience that more cakes were sold in summer than in winter and suspected that this was related to the fact that it is sunnier in the summer months. When building the demand forecasting model for cakes, the bakery selected a number of features such as the sales volumes in previous years, the 30-day moving average, and the amount of rainfall forecasted for the next week.

However, it turned out that rainfall was an unsuitable proxy for “good weather” and hence unsuitable to predict the demand for cakes. In fact, customers bought cakes even in rainy weather. The key driver, in fact, was not the amount of rain but the temperature. More cakes were bought on warm days (even when it was raining) than on cold days (even when it was not raining). One effective method to counter such a measurement bias is exchanging with domain experts who often have a more intuitive feeling about the real cause. A production manager who estimates the expected sales amount every day could have helped to find the “real” proxy for good weather and could have prevented the bakery from choosing a wrong proxy.

5.3. Representation bias: Using apples to predict oranges

The bakery trained a demand forecast model with sales data from city center locations. The algorithm learned the demand patterns and temporal distribution for these specific locations. The same model was then used to create a forecast for train station locations which performed very poorly. The dataset for building the forecast model is not representing the demand distribution as customers in the city center locations and travelers have different consumption patterns. For example, demand in city center locations is highest on weekends.

One effective measure to counter representation bias is using interpretable models for which the relationship between input features and output is transparent and explainable. This would have disclosed the strong positive correlation between the input variable “weekend” and the expected demand for the model that was trained on the city center sales data. Together with consulting domain experts, who know that the train station locations sell more on business days, this mitigation strategy would have helped to demonstrate that the probability distributions differ.

5.4. Label bias: Old wine in new bottles

The bakery relabeled an existing product to address the trend towards healthy nutrition. To do this, it re-invented the existing “organic wheat bread” as “organic wellness bread” and created a new product ID. The promotion campaign was completed and the new product was launched on a Wednesday morning, and the organic wellness bread was positioned as fresh and healthy amongst bakery customers. Label bias would be introduced if employees working on Monday, Tuesday, and Friday were aware of the ID change of the newly launched organic wellness bread, but employees working on Wednesday and Thursday were not.

That is, when purchases were entered into the cash register on Wednesday and Thursday, “organic wellness bread” could be incorrectly labelled as “organic wheat bread” in the data. The demand forecast for “organic wellness bread” would be calculated incorrectly. An effective measure to counter label bias is data plotting. If the data is plotted, changes in the ID of the same product can be detected. On a specific day, the sales of the old ID drops to zero, whereas sales of the new ID has approximately the value that the old ID had before. Moreover, possible spikes in the data that result from human errors can be revealed by data plotting.

5.5. Algorithmic bias: As simple as possible, but no simpler

A newly hired sales manager of the bakery selected a forecasting model that proved successful in her former role in a different industry. She was convinced that a 3-day moving average was simple to use, transparent to understand, and easy to communicate, and hence would serve as a suitable algorithm for forecasting the demand for organic wheat bread. However, given lower demands at the beginning of the week and higher demands towards the weekend, the forecasting algorithm did not account for the complexity of the prediction problem.

This algorithmic bias was costly, as the bakery produced a surplus of units from Mondays to Wednesdays and lost sales from Thursday to Saturday. One effective measure to counter algorithmic bias is rapid prototyping. This way, high deviations between the algorithmic performance in the training data and test data can be detected prior to deploying the model. In the example above, the performance will be poor since it is a case of underfitting. Domain experts can suggest additional input features to increase the complexity of the model.

5.6. Evaluation bias: Benchmarking with caution

The data scientist of the bakery was asked to build a demand forecast model using the sales data from the bakery. Once the prototype for the ML model was developed, a general manager asked for an objective benchmark to develop a better sense of the quality of the prediction model. Because the manager has learned that several online platforms offer free reference datasets, the data scientist is asked to benchmark the developed model with a “neutral” dataset. The data scientist benchmarked the forecasting model against a publicly available bakery sales dataset. Unfortunately, the forecasting model performed poorly and the manager refused to provide a budget for the further development of the ML model.

The data scientist could not demonstrate the value of the forecasting model convincingly because it had been specialized on the proprietary dataset but was evaluated against a public dataset that is not representative for the bakery. To mitigate evaluation bias, the representativeness of the benchmark dataset should have been verified with respect to the relevant population of the bakery. The benchmarking result is only meaningful when the representativeness has been established. Only then does an ML model which performs well on the benchmark dataset also perform well in the context where it is deployed.

5.7. Deployment bias: Stick to the knitting

The marketing manager of the bakery launched a campaign that issued vouchers to a specific regional target group. He built a forecast model that predicted the average spending per visit of a customer. When analyzing the model, he learnt that customers who purchased a coffee had a higher average spending per visit. Using this insight, he launched a new campaign: Some customers receive one free coffee to increase their average spending. However, the campaign did not increase the average spending of the target customers.

The marketing manager assumed that average spending per visit could be increased by providing free coffee. However, by doing this, the prediction model was interpreted incorrectly. The original purpose of the model was to predict average customer spending per visit. Deploying the model for a different decision and justifying the arbitrary human intervention of giving out free coffee represented a misuse of the model. From a statistical perspective, the example illustrates the common fallacy of confusing correlation with causation. That is, there is a correlation between the purchasing of a coffee and spending per visit, but that does not mean that average spending per visit can be increased by providing free coffee.

One effective measure to counter deployment bias is to discuss technical and social consequences. In the example above, discussing how the forecast model that predicts the average spending per visit of a customer can be used when deployed and comparing the purpose of the model with its intended use mitigates the bias. Consequently, the marketing manager is prevented from using the forecast model in an inappropriate context. Another applicable measure is establishing a monitoring plan to track whether the intervention of providing free coffee is effective in increasing the average spending. This would reveal the ineffectiveness of the model.

5.8. Feedback bias: Mind the power of the algorithm

The demand planning team observed a consistent decline in the number of customers and transactions in the train station location over a period of several months. Seasonal and one-off effects could be ruled out, and the product mix remained similar to prior years. The bakery addressed the decline with a promotion to retain existing and to attract new customers: each customer received a free croissant with every coffee-to-go purchased. Although revenues improved slightly, the overall trend could not be stopped.

Only after speaking to travelers who passed by with a coffee and a croissant from a competitor did the bakery become aware of the problem: The competitor provided a free croissant to each customer who gave the bakery a five-star rating on TripAdvisor and GoogleMaps. Accordingly, travelers would find a competing bakery that had many excellent online reviews and go there. As these travelers would also be rewarded with a croissant for five-star ratings, the competitor would further stimulate the positive online rating and generate a reinforcing feedback loop. The competitor thus understood the power of algorithms and found a way to exploit feedback bias to increase recommendations and customer visits.

As a result, it became increasingly difficult for the bakery to counter the competitive attack as the recommendation algorithm was under the control of third-party platform operators. It is also worth noting that the cost for the reward per campaign interaction (i.e., one croissant) was the same for both bakeries. Nevertheless, mitigation methods help to identify and detect such attacks, e.g., by collecting and monitoring recommendations on relevant platforms in competitive markets, and to initiate countermeasures. These could include mobilizing one’s own customer base to generate positive ratings or filing complaints for anticompetitive behavior.

6.1. Contributions to the literature

To date, there is a general scarcity of research on ML bias, despite its high importance. More specific shortcomings include the generic use of the term bias, the inadequately captured link between cause and effect of different biases in the absence of a holistic, comprehensive perspective, and, consequently, challenges in systematically addressing ML bias in marketing through effective mitigation strategies and marketeer education. By addressing these shortcomings we provide several contributions to the literature. First, our comprehensive taxonomy defines the most common biases in ML. Jointly with the conceptual model of the ML application logic, our work provides a focused theoretical account to differentiate eight types of bias. The taxonomy was developed in a stepwise, transparent manner based on a thorough literature analysis. Researchers can use the taxonomy to more effectively study, analyze, and discuss ML bias in other empirical settings, e.g., for further research on organizational practices and processes that generate or address the different types of ML bias, as indicated by Rai (2020).

Second, our results are useful for understanding and analyzing ML bias more systematically, because the developed taxonomy and the respective mitigation strategies provide a holistic view on how ML biases can be identified, avoided, and mitigated along the CRISP-DM development process. CRISP-DM is the most established, and widely accepted standard for managing ML projects. Hence, mapping different bias causes and effects onto CRISP-DM provides a systematic structure for capturing ML bias from a spatial, temporal, and causal perspective. As such, our research addresses Guha et al. (2021, p. 35) call for research by helping to “identify bias […], before much harm is caused”, and responds to prior calls for research on the topic (c.f., Davenport et al., 2020).

Third, the examples from of the bakery chain that contextualized the biases in specific marketing ML applications and the visualizations for each bias should be used in marketing education. Since we focused on the logic and circumstances under which ML bias can occur rather than on the mathematical foundations, the illustrative examples can be used to raise marketeers' awareness of potential ML biases. Sensitizing marketing students and practitioners for the challenges associated with the use of ML in real-world marketing applications is an important strategy to avoid ML bias (Huang and Rust, 2021, Puntoni et al., 2021).

6.2. Managerial implications

Besides the concrete recommended actions to address biases in ML projects in each stage of the CRISP-DM process outlined in Table 2, our work also holds some more general implications for managers when considering ML in marketing, as summarized in Table 3.

There is no bias-free ML. All ML applications have (potential) biases, since otherwise the following assumptions would have to be true. First, the available data would need to perfectly represent the relevant population, including every relevant data feature. Second, all characteristics in the relevant population as well as in the data would need to be socially and normatively desired. Third, there would need to be stability in the relevant population and in the data, in order to prevent any changes. Given that these requirements can hardly be met, all ML applications in marketing may suffer from biases. Thus, AI based on ML does not only demonstrate an opportunity but also a risk for automated marketing activities, and researchers and managers should focus on how to proactively address and mitigate the potential of a social bias, measurement bias, representation bias, label bias, algorithmic bias, evaluation bias, deployment bias, and feedback bias.

There is no panacea for ML biases. Our research demonstrates that ML biases constitute a multifaceted problem that has an impact throughout the entire ML project, including the business understanding, data understanding, data preparation, modeling, evaluation, and deployment phases. ML bias is mostly introduced unintentionally and is relatively difficult to detect at face value. Mitigation methods need to address social, technical, and economic aspects of an ML project. As a consequence, technical mitigation methods should be complemented with non-technical mitigation methods that consider more than merely good performance results. In particular, ML bias removal tools such as “AI Fairness 360” (Bellamy et al., 2018) should be assessed with great caution as the promised outcome can hardly be achieved single-handedly with the tool alone. Given the prevalence of human judgment and human errors in ML, researchers and managers should instead use the full armory of mitigation methods provided.

Change triggers ML biases. ML models are largely data-driven and take over some decision-making agency. However, the context in which the decision-making takes place is not stable, but instead constantly evolves and changes, sometimes gradually and incrementally and sometimes dynamically and radically. We suggest that researchers and managers remain reflective regarding the deployment of their ML models and strive to proactively address adverse effects that can result from changes in the relevant population, its representing data, the ML model, and the deployment context. This is especially important in a dynamic discipline like marketing where consumer behavior, technological possibilities, and the legal context frequently change, making ML predictions based on outdated data harmful for companies.

6.3. Limitations and future research directions

Our literature review of ML biases is necessarily “retrospective”, restricted to existing articles, and subject to the limitations inherent in the original studies. Thus, we advise that future research addresses some of the limitations of our work. It would be insightful to illustrate the biases and the mitigation methods in specific ML projects with real data. By doing so, the contextual conditions under which each of the identified biases can occur would be better captured. Because there is no one-size-fits-all solution to the diverse ML biases, technical and social aspects of ML should be combined to bring context-awareness to research and practice.

Another future research direction is to identify how human biases potentially translate into ML biases. The starting point for our research was to identify ML biases which are known or have been studied in extant research and how these biases can be addressed using effective mitigation strategies. However, for many, if not most, of the identified ML biases, human bias can be the cause. For example, social bias is introduced into the ML context if data that is used for training a model that resembles such human biases. Also, human bias can lead to representation bias, when humans select a non-representative data set for training. Further biases, which might be stemming from human bias, could be identified and contribute to the nascent stream of research on ML bias. Such kind of studies would likely benefit from decades of research particularly in the psychology domain and the provided CRISP-DM framework, as well as the conceptual model provided here. These prior works could serve as a bridge to translate and locate how and when human bias is salient and how it might take effect within the typical logic of ML applications.

6.4. Conclusion

ML can incorporate inadequate properties that lead to both technically incorrect and socially unacceptable results. Besides performance criteria such as reliability, efficiency, and accuracy, addressing bias should be an integral part of any ML application. In contrast to the dominant discussion on bias, which is primarily about discrimination based on race, gender, religion, or membership in a social minority, we also emphasize and illustrate the economic dimension of bias. As such, we show that managing bias in ML projects is not only about fairness, but also about ensuring sustainable economic value from using ML in business settings.